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(54) Optical navigation system 

(57) Optical navigation upon grainy surfaces whose 
orientation is inclined at about 45 0 to the X and Y axes 
of the navigation mechanism is enhanced by: first detect 
that a spatial fitter in use is inappropriate for the orien- 
tation presently occurring, and: second, employ a differ- 
ent and more appropriate spatial filter subsequent to 
such detection. The shape of a correlation surface (89) 
used in the navigation process is tested (93,94) for the 
presence of a ridge in the correlation surface. This gen- 
erates control metrics whose filtered excursions are 
tracked by a control system that changes the spatial fil- 



ter in use. The direction from which illumination arrives 
relative to the X and Y axes affects the range of angles 
(relative to, say, the Y axis) that a filter is effective, by 
changing the apparent size and proportions of the high- 
lights and shadows that are the perceived features ulti- 
mately navigated upon. Stable operation of the control 
system can be enhanced by ensuring overlap of the fil- 
ters' ranges, and can be promoted by dynamically alter- 
ing (102) the direction from which illumination reaches 
the navigation surface(Fig. 14), so that it corresponds 
to, or varies in relation with, the filter in use. 
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Description 

[0001] The present invention relates to an optical navigation system, _ „ 

[0002] The subject matter of this application is related to those of U.S. Patents No. 6,057,540; No. 5,578,813 and 
5 No. 5,786,804. These three patents describe techniques for the tracking of position movement which techniques in- 
clude components of the preferred embodiment described below. 

[0003] Optical navigation upon arbitrary surfaces produces motion signals indicative of relative movement along the 
directions of coordinate axes, and is becoming increasingly prevalent. It is used, for instance, in optical computer mice 
and fingertip tracking devices to replace conventional mice and trackballs for the position control of screen pointers in 
10 windowed user interfaces for computer systems. It has many advantages, among which are the lack of moving parts 
that accumulate dirt and suffer the mechanical wear and tear of use. Another advantage of an.ppfical mouse is that it 
does not need a mouse pad, since it is generally capable of navigating upon arbitrary surfaces, so long as they are 
not optically featureless. 

[0004] Optical navigation operates by tracking the relative displacement of images. A two dimensional view of a 
is portion of the surface is focused upon an array of photo detectors, whose outputs are digitized and stored as a reference 
image in a corresponding array of memory. A brief time later a sample image is also digitized, if there has been no 
motion, then the sample image and the reference image are identical (or very nearly so). What is meant, of course, is 
that the stored arrays appear to match up. If, on the other hand, there has been some motion, then the sample image 
will appear to have shifted within its borders, and the digitized arrays will no longer match (that is, if their borders are 
20 also lined up). The matching process is termed "correlation" and may be performed in various Ways, one of which is 
described in the incorporated Patents. Considered in isolation, correlation answers the narrow question "Are these two 
images aligned?" When the answer is "No," it could be because of intervening motion in any direction, and some 
additional mechanism is needed to find the direction and amount of displacement that will produce correlation results 
of "Yes" or "Almost." What is done is to perform correlations between one of the stored Images (say, the reference 
25 image) and a collection of shifted versions of the other (sample) image. The shifts are no shift, one over, one over and 
one up, one up, one over the other direction, etc., for eight actual shifts and one "null" shift. We would then expect that 
one ofthese nine correlations would be better than all the others, and its direction and amount of shift is taken as an 
indication of the intervening motion. 

[0005] The navigation mechanism maintains velocity (speed and direction) information. When a new sample frame 

so is to be correlated with the reference frame a predicted shift can be used as the starting point for the nine shifts. With 
this arrangement the correlation results contribute to an updated velocity and either ratify or modify the motion just 
predicted. Prediction is used to "pre-shift" the reference frame, until such time as the new sample frame fails to overlap 
a significant portion of the shifted reference frame. At that time a new reference frame is taken. Prediction can be used 
to slow the sample rate down to conserve power (if the light source Is pulsed, or can be turned off between samples) 

35 when the current velocity is low. 

[0006] The basic technique described above works quite well when the surface produces images that appear as 
regions of light and dark defined by closed boundaries. The boundaries need not be abrupt; they can be gradual. And. 
if the shapes of the regions are sufficiently non-symmetrical orirregular, then any closure requirement for the boundaries 
can be relaxed. The shift and correlate mechanism may encounter difficulty, however, when there is a lot of symmetry 

40 and no closed boundaries. Say, for example, a mouse were navigating upon a wood grained desk. Under the right 
circumstances, the images presented to the correlator could appear to represent a series of parallel lines. It is easy to 
detect motion in the direction across the lines, but not in the direction along the lines. The result is an outfoxed mouse 
and a frustrated user. (Grainy surfaces are only sometimes troublesome. Much depends upon feature size, as well as 
orientation. Parallel features whose separation is less than the spacing between photo detectors are not resolved as 

45 /separate entities. Difficult cases arise for parallel features that align with, say, every other (second) or every third photo 
detector. In general, the more the feature separation varies from those troublesome cases, the less difficulty arises. 
Just because a surface is grainy does not mean that it can produce a difficult case.) 

[0007] One solution to this occasional difficulty would be to increase the resolution with which images are represent- 
ed. One might magnify the image more, increase pixel density, or digitize each pixel to a greater number of bits. The 

so hope is that eventually what appears to be regular repeating (and thus indistinguishable) features become irregular 
(and thus distinguishable from one another). Each of these solutions has its own disadvantage, among which are 
increased cost and complexity, and a need for greater speed just to maintain a given rate of tracking over the surface. 
[0008] Another technique.that has been used to mitigate the parallel line problem is (spatial) filtering the images 
before they are correlated. The idea is to modify the arrays to be correlated such that a regular pattern in the data is 

55 suppressed, at least in part. This allows the response of the correlation mechanism to be based more on irregular 
features in the image and less on regular recurring features that produce strong correlations not dependent upon 
motion. The idea is similar to filtering out a known strong signal to recover a weaker one. The filtering is performed by 
altering the values in the various positions of an array according to arithmetic combination with the values in symmet- 
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rically opposed positions disposed around the location of interest. The removal of an interfering signal by common 
mode rejection with a differential amplifier comes to mind. An example will be given in due course, but even at present 
it will be suspected that this technique distorts or alters the image beingtracked. This is acceptable, since we are not 
particularly interested In such fidelity, anyway. We are never going to display to anyone the contents of the array as 
5 an image; we simply want to track it. The prior art contains different spatial filters usable in the manner we have just 
described, and once a suitable one has been identified by the designer it is simply incorporated as a permanent feature 
into the system. 

[0009] One such (spatial) filter has been used in conventional optical mice to deal with the problem of parallel lines 
arising from say, wood grain. It is effective in two orthogonal directions (relative to the spatial pattern with which the 
10 array of photo sensors is laid out), and we shall call it the "Standard" filter. There is, however, a substantial range of 
orientations midway between these two axes that fool the Standard filter, and at which the mouse again begins to 
appear confused if it encounters long parallel features. 
[0010] The present invention seeks to provide improved optical navigation. 

[001 1] According to an aspect of the present invention there is provided a method of spatially filtering an image to 
15 remove grain prior to navigation as specified in claim 1. 

[0012] According to another aspect of the present invention there is provided a method of minimising the effect of 
grain upon navigation as specified in claim 3. 

[0013] A preferred solution to the problem of optically navigating upon grainy surfaces whose orientation is inclined 
at about 45° to the X and Y axes of the navigation mechanism is to: first, detect that the spatial filter in use Is inappropriate 
. 20 for the orientation presently occurring, and: second, employ a different and more appropriate spatial filter subsequent 
to such detection. Two additional filters have been developed that are respectively effective about the 45° and 135° 
inclinations (the trouble regions) of the Standard filter (which is oriented along the X and Y axes). A test has been 
developed for the shape of a correlation surface (or bowl, if viewed from underneath)' used in the navigation process. 
The test uses the presence of a ridge (or saddle) in the correlation surface (bowl), and generates control metrics whose 

2S excursions are tracked by a control system that changes the filter in use. The control system incorporates a time 
constant to prevent thrashing and excessive sensitivity to isolated random variations; The direction from which illumi- 
nation arrives relative to the X and Y axes appears to have an effect on the range of angles (relative to, say, the Y axis) 
that a filter is effective, by changing the apparent size and proportions of the highlights and shadows that are the 
perceived features ultimately navigated upon. For example, if light strikes the navigation surface as rays essentially 

30 parallel to the Y axis (a common practice) then the angular range of operation (for grain orientation) of the Standard 
filter about the X axis may be about half the corresponding range about the Y axis. Similar asymmetries in the ranges 
of operation are noticed in the inclined filters. The operation of the control system can be enhanced by ensuring overlap 
of the ranges of filter operation, lest lack of overlap induce control system instability. Filter range overlap can be pro- 
moted by dynamically altering the direction from which illumination reaches the navigation surface so that it corresponds 

35 to, or varies in relation with, the filter in use. 

[0014] Embodiments of the present invention are described below, by way of example only, with reference to the 
accompanying drawings, in which: 

Figure 1 is a depiction of a twenty-five cell digitized image for an optical navigation system that is stored in memory 
to as a two dimensional data structure; 

Figure 2 is an image after the fashion of Figure 1 , but that has been artificially enlarged to facilitate filtering; 
Figure 3 is an illustration of a "Standard" filter used in the prior art but that causes problems when diagonal grain 
is encountered; 

Figure 4 illustrates the arithmetic outcome of filtering an enlarged image of Figure 2 with the filter of Figure 3; 
45 Figure 5 is an example showing a moving input image against horizontal grain and its usable filtered results; 

Figure 6 is an example showing the same moving input image as in Figure 6, but against diagonal grain, and 
illustrates the production of unusable results from the Standard filter; 

Figure 7 is an illustration of a filter that can be used under certain circumstances when the Standard filter of Figure 
3 fails; 

so Figure 8 is an illustration of another filter that can be used under certain other circumstances when the Standard 

filter of Figure 3 fails ; 

Figure 9 is an example showing the moving input image of Figure 6 but filtered with the filter of Figure 7 to produce 
usable results ; 

Figure 10 is an example similar to Figure 9, but using the filter of Figure 8 ; 
55 Figure 11 is an illustration of a sample correlation surface produced from an array of correlation values; 

Figure 12 is a state diagram illustrating a filter selection process based on the shape of a correlation surface that 
may be followed during optical navigation to avoid the mischief caused by a grain unfavorable to a filter in use; 
Figure 13 is state diagram similar to that of Figure 12, but incorporating a change in illumination location that is 
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controlled by an examination of the shape of a correlation surface; and 

Figure 14 is a diagram showing the locations of sources of illumination selectable by analysis of correlation results. 

[001 5] Before we undertake an explanation of the preferred solution to the problem, it will be convenient to demon- 
5 strate the problem. In the course of that we shall present some simplified examples that will serve as. good "before" 
and "after" illustrations, and further assist us by providing a useful notation that we may use in subsequent discourse. 
Definite examples will also remove some of the aura of abstraction that might otherwise contaminate the desired un- 
derstanding. 

[0016] Refer now to Figure 1, wherein is depicted a five pixel (X=1 to 5) by five pixel (Y=1 to 5) image array (1) of 

10 twenty-five digitized values (a-y) obtained from an array of photo sensors (not shown). The digitized image array 1 
may be thought of as being twenty-five six-bit (or perhaps eight-bit) values stored in a memory data structure organized 
as a two dimensional array. In a preferred optical navigation circuit for a mouse or other pointing input device for a 
computer, the actual digitized image would preferably be sixteen by sixteen, or perhaps eighteen by eighteen. We 
choose to proceed in this explanation with five by five, not so much for intellectual simplification (the same issues 

f 5 arise), but because the size of the actual case is inconveniently large, both in terms of drawing real estate and in terms 
of notation consumed, without adding any real benefit over a smaDer rendition such as the five by five case we show. 
[0017] To proceed then, Figure 1 represents an optical image that has been focused on an array of photo detectors 
(not shown), digitized and then stored in memory. Further, it might be an image that has just been acquired (a sample 
image) or it might be one that has been obtained some time ago and is being used as a reference image. 

20 [0018] Both sample and reference images are (spatially) filtered before they are used by the correlation process. 
Since the images are of finite extent, we can expect any filtering process to exhibit some transient-like behavior at the 
edges of the image. To minimize the effect of this on the correlation mechanism, we first expand the images to be 
filtered. How this is done is shown in Figure 2. Note that the expansion moves the outer edges of the array out one 
row or column, in each direction as appropriate, with the original four corners each being replicated three times. Thus, 

25 array 1 of Figure 1 becomes expanded array 2 in Figure 2, and has seven rows and seven columns. Observe how, for 
example, the original value 'a' (3) is replicated in the expansion as three more "a's" (4, 5 and 6). The same thing happens 
to 'e', V and y . in the other corners. 

[0019] Notice also that Figure 2 depicts twenty-five locations (the Cartesian product of five X positions called X1 
through X5 and five Y positions called Y1 through Y5). Each of these twenty-five locations defines a nine member 
30 (three by three) ordered collection that will be an instance of input to the filter. The filter operates on the ordered nine 
members (which are multi-bit numbers) and produces a single numerical output that we will associated with the center 
of the three by three input. 

[0020] It should not be assumed that only three by three filters are appropriate. Other filter sizes may be used, and 
they might also be asymmetrical (e.g., three by four). 

35 [0021] The filter we shall term the "Standard" filter is shown in Figure 3. it is an array 7 of nine coefficients that is 
positioned over the enlarged image in each oflhe twenty-five filter positions indicated by the X(1-5) by Y(1-5) indices. 
At each of these locations it produces a single output value according to the summations of the cells contributing as 
multiplied by the coefficients in the filter. The array 8 in Figure 4 indicates exactly what five-by-five filtered array will 
result from application ofthe Standard filter 7 in Figure 3 to the expanded array 2 of Figure 2. 

40 [0022] There is one additional filter property that we need to mention, however. If the digitized inputs were, say, eight- 
bit values then according to the arithmetic indicated in Figure 4 we might need ten-bit values to record the results, if 
we were to faithfully preserve the resulting arithmetic values. We could do that, but we dont. Instead, we take another 
approach. While we want to correlate images with their shifted versions, we find that we can do that as readily by 
responding only to the "shape" of the image, as by giving importance to the "shading" of the image. We accomplish 

45 this by taking only the sign bit of the filter's numerical result as the output of the filter. (In another embodiment, one 
might choose to take some intermediate number of bits.) In taking only the sign bit we may sacrifice a small amount 
of certainty from time to time, but we gain a tremendous benefit in terms of reduced circuit complexity, reduced size 
and cost ofthe optical navigation IC, perhaps even increased speed of operation for a given clock rate. Accordingly, 
we fabricate the filter to simply output the one bit sign of the actual arithmetic result for those operations associated 

so with the filter and that are indicated in the various cells of Figure 4. 'With this arrangement (which we shall use in the 
figures that follow), a one as a filter output indicates that the numerical result for that cell was negative, while a zero 
indicates that it was zero or positive. 

[0023] Now refer to Figure 5, where we illustrate a simplified prior art case where the Standard filter behaves in a 
suitable manner. On the left-hand side of Figure 5 are three input images 9. 14 and 19 that a sequence of mouse 
55 movements over an unchanging image, as indicated by arrows 24 and 25. The movements are separated by one pixel 
"down" (image 9 to 14) and then by one pixel "over" (image 14 to 19). The images include a distinguishable feature 
that is a square of four adjacent 1's (shown for convenience only in heavy boxes 10, 15 and 20 in images 9, 14 and 
19, respectively). These unfiltered input images (9, 14 and 19) include a horizontal grain component, which was added 
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to show that it is filtered out, as it is supposed to be. That horizontal grain component is the top row of 1's in 9, which 
then becomes next row down from the top in input images 14 and 19. 

[0024] Before proceeding we need to identify a potentially confusing simplification that will be present from Figure 
5 onward. We have said that the values that populate the input arrays 9, 14 and 19 are multi-bit values. For the sake 

5 of simplicity in our figures from Figure 5 on, however, we will pretend that we need represent only the conditions of 
'light" and "dark", which we will indicate with the one-bit symbols "1 " and "0", respectively. We could have used "L" 
and "D" instead, but then somebody would complain "Hey, how do I do arithmetic on L and D ....?". It may help to think 
of the "1's" (L's) as being the larger multi-bit (integer) numbers, while the "0's" (D's) are the smaller numbers, and that 
the filter incorporates some type of AC coupling. Alternatively, one can think of the input image array as having been 

10 digitized to one bit: one for light and zero for dark. We don't really do that in the actual part, but the simplification is 
valid, and it is useful in our examples. Just don't forget (and this is the potential for confusion) that on the output side 
of the filter we have a sign bit, where one means negative and zero non-negative, which may seem at variance with 
the way the inputs are described. And while the output notation amounts to a desirable simplification, it is an actual 
one that we do use in the real parts. That said, however, we are never going to take advantage of the idea that "This 

15 number is negative, and therefore As explained above (shape versus shading), the ones and zeros in the filter 
output are just the two symbols in a minimal description. 

10025] In summary, the filter is simply an operational combinatorial rule of arithmetic that is applied to a symbol set 
on the input image. It maps nine adjacent input values into one value in the filtered output. Many different adjacent 
filter positions are used to create the filtered output image. And while one can compare neighbor symbols in the input 

20 or in the output, it is not fair to think that an input symbol (say, a "1") is an indivisible nuclear entity that passes untrans- 
formed through the filter and simply comes out the same kind of thing that it was, but just in a different place. We may 
agree that fundamentally they are all numbers, but what they mean before and after filtering are not the same things. 
And we must remember that our use of only single "1's" and single "0's" on the input side of the filter (which is unfor- 
tunately similar to the "1's" and "0's" on the output side) is strictly a useful convenience (in place of multi-bit values), 

25 once properly understood. 

[0026] To resume, then, note how arrows 26, 27 and 28 relate the filtered outputs (11, 16 and 21) to their respective 
unfiltered inputs (9, 14 and 19). We note three important things. First, the horizontal grain of the inputs is removed. 
Well, that is, in part, what the Standard filter is for. Second, note how input features 10, 15 and 20 (which are really 
the same thing, just in a different location in the input image, owing to motion of the mouse), are matched by corre- 

30 sponding output features 12/13, 17/18 and 22/23. Now, it is true that feature 12/13 In the filtered output image 11 
scarcely matches the original input feature 10 in unfiltered input image 9; in fact, we might say that the feature 10 was 
deformed (went "splat") when it went through the filter. That's okay, since in every case, the feature was deformed 
(went "splat" ) in the same way. What is more, and this is the third important thing to note, the locations in the filtered 
images 11.16 and 22 of the deformed features varies proportionally (exactly one to one) as the variation of the corre- 

35 sponding input feature in the unfiltered images. It is this well behaved correspondence that allows the correlator to 
track the deformed image as readily as if it were not deformed. 

[0027] Now take a look at Figure 6. It is again a sequence of three unfiltered input images 29, 36 and 43, with one 
pixel of "down" motion (arrow 35) followed by one pixel of "over" motion (arrow 42). That Is, we note that the mouse 
motion for Figure 5 is the same as for Figure 6. Also as before, we have (the same!) input feature of four adjacent ones 
40 (shown in heavily outlined squares 30, 37 and 44). In this case, however, there is a mild diagonal grain as "background" 
(running from upper left to lower right). 

[0028] But now notice the filtered outputs 31, 38 and 45, whose correspondence to their unfiltered counterparts is 
indicated by arrows 34, 41 and 48, respectively. The input feature (30, 37, 44) still goes "splat", but not in a well behaved 
way. The orientation of 32/33 is different from that of 39/40, and seemingly inconsistent with that of 46/47. The output 

45 feature 32/33 etc. moves in a way that does not correspond to movement of the input feature 30. (We note also the 
unexpected different appearances of 1 's 49, 50 and 51 . These, however, are probably artifacts of the expansion at the 
image boundary and of the reduced size of our example, and can probably be ignored rather than taken as evidence 
of filter misbehavior.) All in all, that pesky Standard filter 7 of Figure 3 is fickle, and has left us in the lurch! All of this 
sends the correlator out to lunch (it gets terminally confused). This disgusting situation is what we have set out to fix. 

so [0029] Referring now to Figures 7 and 8, there are shown two filters 52 and 53 (respectively) that may be used in 
circumstances when the Standard filter 7 of Figure 3 causes problems owing to diagonal grain in the unfiltered input 

[0030] Without further ado, let's see what these filters do with the situation that out-foxed the Standard filter and 
flummoxed the correlator, Refer then, to Figure 9, where we start just as in Figure 6 (where the Standard filter failed) 
55 but apply the filter Ql instead. As before, we have three input images 54, 55 and 56 accompanied by input features 
65, 66 and 67 (all of which are the same as 29, 36, 33 and 30, 37.44 of Figure 6). There is even the same cursor motion 
of one down (arrow 63) and one over (arrow 64) as in Figure 6. Arrows 60, 61 and 62 relate the unfiltered input images 
to filtered images 57, 58 and 59, respectively. As feature 65 (four adjacent 1's in a square pattern) goes through filter 
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Ql it still goes "splat", and comes out a squished and crooked pattern 68 that does not even have the same number 
of Vs. The filter Ql is well behaved where it counts, however. Note that patterns 96 and 70 are similarly squished and 
crooked, and that their positions track that of their originating features in the unfiltered images. Filter Ql 52 of Figure 
7 works where the Standard filter failed. 

5 [0031] Before proceeding further, another note on notation is in order. Suppose that we knew that a parallel grain 
was present, and we arranged for it to appear as vertical lines. Looking at it, we could say that the top was twelve 
o'clock, and that were other positions as well, such as three o'clock and nine o'clock. Let us make the following angular 
correspondences: twelve o'clock is 0°, three o'clock is 90°, six o'clock is 180°, nine o'clock is 270% and so forth, with 
all the other values lying in the obvious way between those we have mentioned. Now ask the predominant direction 

to (if there is one) of a feature in the image, and give it a di rection in degrees according to the preceding convention. The 
first quadrant (Ql) is, of course, the angles zero through ninety degrees, and has a reflection in the third quadrant. The 
second quadrant (Qll) is, of course, the range of ninety through one hundred eighty degrees, and has a reflection in 
the fourth quadrant. The filter 52 shown in Figure 7 is called "Filter Ql" because it appears to work best on features 
whose axes lie approximately centered in the first and third quadrants, when the above conventions are followed. So, 

15 for example, look at Figure 9 and identify the diagonal grain running from upper left to lower right. Now rotate the figure 
clockwise about 45°so that the grain appears to be vertical. Now notice that the major axis of the clump of 1's (there 
are eight that are adjacent) appears to be in the first quadrant. The filter Ql works better on this example than does its 
companion Qll. The filter 53 of Figure 8 is called "Filter Qll" because it appears to work best on features whose axes 
lie approximately centered in the second and fourth quadrants. 

20 [0032] Figure 10 is an example similar to that of Figure 6, where we expect the Standard filter to faii, but that the 
filter Qll 53 functions. In that figure we see that there are again three unfiltered image inputs 71 , 72 and 73, that each 
contain an image feature (77, 78 and 79, respectively). In this case, however, the diagonal grain runs form upper right 
to lower left, and the feature (four adjacent 1 's highlighted in heavy boxes 77, 78 and 79) starts out in a slightly different 
location and moves along a slightly different path. Arrow 86 indicates that it moves down one pixel from image 71 to 

25 become image 72, while arrow 87 indicates that image 72 moves over one pixel to the left to become image 73 (previous 
examples moved to the right). The correspondence between unfiltered input (71, 72 and 73) and filtered output (74, 
75 and 76) is indicated by arrows 83, 84 and 85, in the same manner as for other figures. Likewise, note the regularly 
moving and well behaved (even if they did go "splat") output featuresSO, 81 and 82. They will not confuse the correlator, 
and can be navigated upon. 

30 [0033] To be sure, these filters Ql 52andQII 53 are not, even between them both, able to entirely replace the Standard 
filter 7 of Figure 3. That is, they will each fail if used in unfavorable circumstances, just as the Standard filter will. 
Furthermore, the Standard filter works quite well along the coordinate axes (as defined above, which is to say, for 
vertical and horizontal grain). But between these three filters we do have complete coverage. What we need then, is 
a way to use the different filters in those circumstances that they are each best suited for. 

35 [0034] A tool has been developed that predicts the best filter to use as the mouse navigates over the surface. To 
explain what it is and how it works we first need to say a bit more about the next steps in the optical navigation process: 
prediction and correlation. Each of these concepts is relatively easy to appreciate in isolation, and are discussed at 
some length in the above-mentioned patents. We wont dwejl on them at any great length here, mentioning only what 
is needed to proceed. 

40 [0035] The optical navigation system keeps track of its velocity (speed and direction), and uses that information to 
predict where the next sample frame will probably fall on the reference frame in use. That is to say, where on the 
(filtered) reference frame should the next (filtered) sample frame be centered prior to correlation. 
[0036] Correlation involves the making of nine (or perhaps more, but nine is typical) comparisons between the nine 
trial sample frame positions and the reference frame, with the idea being that at one of those positions the images will 

45 line-up or match better than at any of the others. The nine positions are: no change (use the prediction unaltered); one 
pixel up; one pixel up and one to the left; one up and one to the right; one to the left; one to the right; one down; one 
down and one to the left; and, one down and one to the right. In the system we have set out herein, with the filter 
outputs being sign bits only, the comparison process at each trial position for correlation is relatively easy. At each trial 
position, we simply count the number of differences (or alternatively, the number of 1's) between corresponding loca- 

50 tions for the two images. The result is nine numbers that we may arrange in an array corresponding to the different 
correlation shifts used to find them. The effect to describe a correlation surface, which, under favorable circumstances, 
has certain properties. Say, for example, that the surface we are navigating upon is free of grain or any other poisonous 
artifacts, that mouse velocity is well within limits, and that everything is generally operating as it should. We then expect 
that the last prediction would be correct, and that the peak of the correlation surface would be at the center cell of the 

55 nine, with the remaining eight cells all having substantially smaller values. This is the kind of situation depicted in Figure 
11 , where the array 88 of nine correlation values (CS1 through CS9), if projected as being surface elevations, would 
produce a peaked correlation surface 89. Some writers have viewed the correlation surface from "underneath" as 
opposed to "from above" as we are doing here. When viewed from below a well formed correlation surface resembles 



EP 1 255 219 A1 



a "bowl" and is sometimes so called. 

[0037] (A slight digression is in order to keep this simplified discussion simple while at the same time keeping it 
honest. Clearly, a five by five sample frame is not adequate in a system that uses prediction and correlation as described. 
How could we have a reference frame of such small size and predict the next sample to correlate three over and four 

s up? There would not be any overlap Within which to correlate! So we appreciate the utility of larger sample frames. But 
even so, as we shift to do prediction we guarantee that portions of the sample frame will not overlap the reference 
frame (and will therefore not contribute), and what is worse, for some trial shifts there will be greater potential overlap 
of the images than for others. What this means is that, unless we do something to correct it, the nine correlation numbers 
will not all have the same units, and cannot properly be directly compared to one another. That is, for trial shifts where 

10 there is greater mis-registration between the frame boundaries, there are simply fewer pixel locations in the first place 
to count toward a good match of the images. So, the correlation process "normalizes" the correlation numbers so that 
they all have the same weight despite the varying degrees of frame boundary mis-registration associated with their 
individual calculation circumstances. It is these normalized numbers that we use for the correlation surface, and which 
drive the navigation process.) 

is [0038] The tool we shall use to predict the best filter to use analyzes the shape of the correlation surface. The tool 
will detect when the correlation surface 89 develops ridges, instead of having an isolated peak. We note that ridges 
could appear in two ways: aligned with the axes and diagonally. With that in mind, we define the following two metrics: 

20 (l) AR = ^"{CS8 -CS4 + CS2 - CS6} 



(2) DR=JH CS7-CS9 + CS3 -CS1} 

25 [0039] The metric AR (for Axial Ridge) indicates presence of a ridge that runs In a direction parallel with one of the 
axes. If the absolute value of AR gets above a certain threshold (say, 15% or 20% of the maximum value of a cell in 
the correlation array 88). then we become suspicious of how well the filter in use is performing. The "raw" component 
of AR (the sum of the two differences within the braces) is filtered by an exponential or auto-regression mechanism (.7) 
to retard its rate of change by a suitable time constant Furthermore, only images whose separations in the X and Y 

30 axes are small enough to imply a good correlation surface are used to calculate AR. The idea is that AR will be used 
as input to a servo-control mechanism, the image data is apt to be somewhat noisy, and we don't want to. the servo to 
be too easily "spooked" by an isolated random excursion in the data. The other metric DR (Diagonal Ridge) indicates 
the presence of a ridge that runs in a direction diagonal with the axes, and is calculated with restricted input and filtered 
in the same manner as Is AR. It, too, has an associated threshold for creating suspicion. And while we could steer an 

35 image filter selection mechanism from these two metrics considered individually, we prefer to combine them into one 
indicator, thus: 

RM=V(AR 2 + DR 2 ) (3) 

40 

[0040] RM (Ridge Metric) will get larger as either of AR or DR get larger in magnitude, and although there are other 
ways to take note of the individual values of AR and DR, RM is a convenient way to unify a common meaning of those 
two metrics into a single indicator. We are not yet finished with AR and DR, however. We note that the signs of AR and 
DR contain useful information. If we assume that in a system of the sort we have been describing that a correlation 

45 surface can get ridges (or the bowl -as viewed from below - can get troughs), then we can also expect that a properly 
operating system will not experience, after suitable filtering, non-transient troughs in its correlation surface (or ridges 
in a bowl). We decline to offer a rigorous proof of this, but it is indeed the case, and navigation by correlation depends 
upon it. Anyhow, it follows that if (and this is how our angular measure convention was stated) the Y axis is in the CS2 
- CS5 - CS8 direction, then AR>0 implies that there is a ridge along that axis. If AR<0 it implies that there is a ridge 

50 along the X axis. The sign of DR can likewise be interpreted to distinguish between the two possible diagonal ridges. 
[0041] Refer now to Figure 1 2, which is a state diagram 90 of how an optical navigation system, such as for a mouse, 
and using the techniques set out herein, may be made less susceptible to the mischief caused by grain in the surface 
to be navigated upon. At the conclusion of certain initial conditions (turn on, reset, etc.) a transition START 91 enters 
a state 92 NAVIGATE WITH STANDARD FILTER. While in this state the optical navigation system operates mostly in 

55 a conventional manner and uses the Standard Filter 7, except that it is calculating and monitoring AR, DR and RM. 
The transition 93 from state 92 to itself indicates this for situations where RM has a value that indicates an absence 
of ridges in the con-elation surface 89. 
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[0042] Now suppose that the value of RM changes to indicate the emergence of a ridge in the correlation surface 
89. Since it is the Standard filter 7 that is presently in use, we may assume that it is a diagonal ridge that is appearing, 
as we know the Standard filter is effective in removing axial ridges. There are two ways that a diagonal ridge can 
appear, however (upper left to tower right, and upper right to lower left), and the filter Ql 52 has, in one embodiment, 
been found to work best for upper left to lower right, while the filter Qil 53 has been found to work best for diagonal 
grain going from upper right to lower left. The difference between the two cases is indicated by the sign of DR, as 
explained previously. Accordingly, if RM is high enough and DR<0, then transition 94 leads to the state 95 NAVIGATE 
WITH FILTER Qll. Transition 100 keeps state 95 in effect as long as DR<0 remains the case. If DR changes sign to 
DR>0, however, transition 98 puts state 92 NAVIGATE WITH STANDARD FILTER back into effect. 
[0043] On the other hand, if while in state 92 RM is high enough and DR^O, then transition 96 leads to the state 97 
NAVIGATE WITH FILTER Ql. Transition 101 keeps state 97 in effect as long as DR^O remains the case. tfDR changes 
sign to DR<0, however, transition 99 puts state 92 NAVIGATE WITH STANDARD FILTER back into effect. 
[0044] Now, some remarks are in order. It will be recalled that earlier we gave a convention for describing angular 
orientation of navigable features relative to the direction of the grain, It can also be used to describe the positions of 
rotated grain for a given filter, or for different filters, relative to an original condition. 

[0045] In keeping with earlier teachings, it is desirable that the light source create highlights and shadows on the 
surface to be navigated upon. This is best done by creating a low or"grazing" angle of incidence for the light from the 
source of illumination. In the embodiment in which the fitters Ql and Qll were developed and used the source of illu- 
mination was located at (according to our angular convention) 180°. The ranges of filter effectiveness that resulted 
overlapped each other, and were somewhat larger than the selected operating ranges shown in the table below: 



Standard Filter 


3200-40°. 85° - 95°, 140° - 220°, 265° - 275° 


Filter Ql 


40° -85°. 220" -265° 


Filter Qll 


95° - 140°, 275° - 320° 



[0046] It can be seen from the above table that the Standard filter has different ranges of operation about the X axis 
90° than about the Y axis. It appears that the wider range of operation abut the Y axis arises from the visibility ofthe 
grain being minimized by end-on illumination, compared to the X axis case where the visibility ofthe grain is maximized 
by broadside illumination. Coping with the X axis case is left entirely up to the filter, whereas in the Y axis case the 
manner of illumination assists the filter. 

[0047] We note that neither filter Ql nor filter Qll is centered over either of the diagonals 45° nor 135°, or over their 
extensions of 225° and 31 5°. It appears that the reason for this asymmetry lies more in the nature of the filters Ql and 
Qll themselves. Nevertheless, they cooperate well with the Standard filter by operating in the regions not covered by 
the Standard filter. For example, the Standard filter does not operate from 45° to 80°. but that region is covered by the 
40° to 85° section of filter Ql . This example is one of four such instances. 

[0048] Now, to return to the algorithm expressed by the state diagram 90 of Figure 12, if one of states 95 or 97 is in 
effect, then there is a diagonal grain present, and what is more, as between the two states, they each represent max- 
imally different diagonal grains. And while it might happen somehow that the grain for each state gets worse (DR gets 
more negative for state 95, or more positive for state 97), we find it difficult to imagine how it could abruptly go from 
one type of diagonal to the other (DR changes sign). SOoo, if DR does change sign, the safest thing to do is to get 
back to the Standard filter, because if the direction of the grain is changing significantly (or a perverse user is rotating 
the mouse), it is reasonable to expect it to pass next through a region covered by the Standard filter. Any other ar- 
rangement would risk loop instability. 

[0049] Now, there is yet another method for reducing the effects of unfavorable grain orientation tp the filter, and that 
is to dynamically alter the location of the source of illumination, so as to illuminate the grain along the direction of its 
axis. In circumstances where the grain is essentially groves and their intervening shoulders, this can have the effect 
of minimizing the long stringy shadows (and their adjacent parallel highlights) that reveal the grain during broadside 
illumination. So, in some circumstances we could alter the direction on illumination instead of changing filters. 
[0050] It is true that changing illuminators would probably require a brief training period for setting the level of illu- 
mination and AGC, etc.. but these need only a short time and the optical navigation runs a pretty high rate (and with 
a low enough duty cycle) that this extra overhead would not be objectionable. Also, no increase in power consumption 
need be accompanied by this strategy. 

[0051] The state diagram 102 of Figure 13 illustrates how this can be implemented, assuming that the different 
locations of illumination shown in Figure 14 are available. If there were more cells than nine in the correlation surface, 
then greater granularity than just two kinds of axial granularity and two kinds of diagonal granularity could be observed, 
and a more refined response in terms of a selected location for illumination would be possible. 
[0052] Lastly, it may be preferable to combine the two strategies, so that the filter in use is changed and the location 
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of the source of illumination is altered, as well. Suppose, for example, that the surface being navigated upon has a 
grain that, instead of being actual groves, is a pattern of discrete holes or projections whose highlights and shadows 
combine to synthesise a grain. The discrete nature of the hole or projections would mean that they continue to create 
highlights and shadows (they remain visible as grain) regardless of the direction of illumination. The amount of grain 
s in the image presented to the spatial filter would thus vary only slightly as a function of the direction of illumination. 
[0053] The disclosures in United States patent application no. 09/845,544, from which this application claims priority, 
and in the abstract accompanying this application are incorporated herein by reference. 



1. A method of spatially filtering an image to remove grain prior to navigation in an optical navigation system, the 
method including the steps of: 

15 (a) navigating (92) with a first filter; 

(b) while navigating according to step (a), generating an ordered collection (88) of correlation values; 

(c) examining the ordered collection of correlation values generated in step (b) for the presence of a ridge in 
a correlation surface (89) described by the values in the ordered collection; 

(d) if the examination in step (c) reveals a ridge, then commencing (94,96) to navigate with a second filter in 
20 place of the first filter; 

(e) while navigating (95,97) according to step (d), examining an ordered collection of correlation values pro- 
duced while the second filter is in use; and 

(e1) continuing to navigate (100,101) according to step (d) while the examination in step (e) reveals the 
25 absence of a ridge; and 

(e2) if the examination in step (e) reveals a ridge, then re-instituting (98,99) the use of the first filter, and 
resuming navigation according to step (a). 

2. A method as in claim 1 wherein the second filter comprises a selected one of third and fourth filters (52, 53) that 
30 are spatially oriented at 90° to one another and step (c) includes the step of determining which of the third and 

fourth filters most effectively deals with the revealed ridge. 

3. A method of minimising the effect of grain upon navigation in an optical navigation system, the method including 
the steps of: 

35 

(a) navigating (103) with a first source of illumination in a first location; 

(b) while navigating according to step (a), generating an ordered collection (88) of correlation values; 

(c) examining the ordered collection of correlation values generated in step (b) for the presence of a ridge in 
a correlation surface (89) described by the values in the ordered collection; 

40 (d) if the examination in step (c) reveals a ridge, then extinguishing the first source of illumination and com- 

mencing (104) to navigate with a second source of illumination in a second location; 
(e) while navigating ( 1 05) according to step (d), examining an ordered collection of correlation values produced 
while the second source of illumination is in use; and 

45 (e1 ) continuing (1 06) to navigate according to step (d) while the examination in step (e) reveals the absence 

of a ridge; and 

(e2) if the examination in step (e) reveals a ridge, then (107) extinguishing the second source of illumina- 
tion, re-instituting the use of the first source of illumination, and resuming navigation according to step (a). 

so 4. A method as in claim 3 wherein the first and second sources of illumination are selected to maximize the end-on ( 
illumination of the revealed ridge. 

5. An optical navigation system including apparatus means operable to perform the steps of a method according to 
any preceding claim. 

55 
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